Computational Emotion Analysis From Images: Recent Advances and Future Directions

Emotions are usually evoked in humans by images. Recently, extensive research efforts have been dedicated to understanding the emotions of images. In this chapter, we aim to introduce image emotion analysis (IEA) from a computational perspective with the focus on summarizing recent advances and suggesting future directions. We begin with commonly used emotion representation models from psychology. We then define the key computational problems that the researchers have been trying to solve and provide supervised frameworks that are generally used for different IEA tasks. After the introduction of major challenges in IEA, we present some representative methods on emotion feature extraction, supervised classifier learning, and domain adaptation. Furthermore, we introduce available datasets for evaluation and summarize some main results. Finally, we discuss some open questions and future directions that researchers can pursue.Comment: Accepted chapter in the book "Human Perception of Visual Information Psychological and Computational Perspective

Affective Image Content Analysis: Two Decades Review and New Perspectives

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.Comment: Accepted by IEEE TPAM

APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval

With the popularity of social media, an increasing number of people are accustomed to expressing their feelings and emotions online using images and videos. An emotion-based image retrieval (EBIR) system is useful for obtaining visual contents with desired emotions from a massive repository. Existing EBIR methods mainly focus on modeling the global characteristics of visual content without considering the crucial role of informative regions of interest in conveying emotions. Further, they ignore the hierarchical relationships between coarse polarities and fine categories of emotions. In this paper, we design an attention-aware polarity-sensitive embedding (APSE) network to address these issues. First, we develop a hierarchical attention mechanism to automatically discover and model the informative regions of interest. Specifically, both polarity-and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we propose a generated emotion-pair (GEP) loss to simultaneously consider the inter-and intra-polarity relationships of the emotion labels. Moreover, we adaptively generate negative examples of different hard levels in the feature space guided by the attention module to further improve the performance of feature embedding. Extensive experiments on four popular benchmark datasets demonstrate that the proposed APSE method outperforms the state-of-the-art EBIR approaches by a large margin

PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression

Existing methods on visual emotion analysis mainly focus on coarse-grained emotion classification, i.e. assigning an image with a dominant discrete emotion category. However, these methods cannot well reflect the complexity and subtlety of emotions. In this paper, we study the fine-grained regression problem of visual emotions based on convolutional neural networks (CNNs). Specifically, we develop a Polarity-consistent Deep Attention Network (PDANet), a novel network architecture that integrates attention into a CNN with an emotion polarity constraint. First, we propose to incorporate both spatial and channel-wise attentions into a CNN for visual emotion regression, which jointly considers the local spatial connectivity patterns along each channel and the interdependency between different channels. Second, we design a novel regression loss, i.e. polarity-consistent regression (PCR) loss, based on the weakly supervised emotion polarity to guide the attention generation. By optimizing the PCR loss, PDANet can generate a polarity preserved attention map and thus improve the emotion regression performance. Extensive experiments are conducted on the IAPS, NAPS, and EMOTIC datasets, and the results demonstrate that the proposed PDANet outperforms the state-of-the-art approaches by a large margin for fine-grained visual emotion regression. Our source code is released at: https://github.com/ZizhouJia/PDANet.Comment: Accepted by ACM Multimedia 201

Age-related decline in hippocampal tyrosine phosphatase PTPRO is a mechanistic factor in chemotherapy-related cognitive impairment.

Chemotherapy-related cognitive impairment (CRCI) or chemo brain is a devastating neurotoxic sequela of cancer-related treatments, especially for the elderly individuals. Here we show that PTPRO, a tyrosine phosphatase, is highly enriched in the hippocampus, and its level is tightly associated with neurocognitive function but declined significantly during aging. To understand the protective role of PTPRO in CRCI, a mouse model was generated by treating Ptpro-/- female mice with doxorubicin (DOX) because Ptpro-/- female mice are more vulnerable to DOX, showing cognitive impairments and neurodegeneration. By analyzing PTPRO substrates that are neurocognition-associated tyrosine kinases, we found that SRC and EPHA4 are highly phosphorylated/activated in the hippocampi of Ptpro-/- female mice, with increased sensitivity to DOX-induced CRCI. On the other hand, restoration of PTPRO in the hippocampal CA3 region significantly ameliorate CRCI in Ptpro-/- female mice. In addition, we found that the plant alkaloid berberine (BBR) is capable of ameliorating CRCI in aged female mice by upregulating hippocampal PTPRO. Mechanistically, BBR upregulates PTPRO by downregulating miR-25-3p, which directly targeted PTPRO. These findings collectively demonstrate the protective role of hippocampal PTPRO against CRCI

Synergistic Effect of SRY and Its Direct Target, WDR5, on Sox9 Expression

SRY is a sex-determining gene that encodes a transcription factor, which triggers male development in most mammals. The molecular mechanism of SRY action in testis determination is, however, poorly understood. In this study, we demonstrate that WDR5, which encodes a WD-40 repeat protein, is a direct target of SRY. EMSA experiments and ChIP assays showed that SRY could bind to the WDR5 gene promoter directly. Overexpression of SRY in LNCaP cells significantly increased WDR5 expression concurrent with histone H3K4 methylation on the WDR5 promoter. To specifically address whether SRY contributes to WDR5 regulation, we introduced a 4-hydroxy-tamoxifen-inducible SRY allele into LNCaP cells. Conditional SRY expression triggered enrichment of SRY on the WDR5 promoter resulting in induction of WDR5 transcription. We found that WDR5 was self regulating through a positive feedback loop. WDR5 and SRY interacted and were colocalized in cells. In addition, the interaction of WDR5 with SRY resulted in activation of Sox9 while repressing the expression of β-catenin. These results suggest that, in conjunction with SRY, WDR5 plays an important role in sex determination

ER Stress Negatively Modulates the Expression of the miR-199a/214 Cluster to Regulates Tumor Survival and Progression in Human Hepatocellular Cancer

Background: Recent studies have emphasized causative links between microRNAs (miRNAs) deregulation and tumor development. In hepatocellular carcinoma (HCC), more and more miRNAs were identified as diagnostic and prognostic cancer biomarkers, as well as additional therapeutic tools. This study aimed to investigate the functional significance and regulatory mechanism of the miR-199a2/214 cluster in HCC progression. Methods and Findings: In this study, we showed that miR-214, as well as miR-199a-3p and miR-199a-5p levels were significantly reduced in the majority of examined 23 HCC tissues and HepG2 and SMMC-7721 cell lines, compared with their nontumor counterparts. To further explore the role of miR-214 in hepatocarcinogenesis, we disclosed that the ER stressinduced pro-survival factor XBP-1 is a target of miR-214 by using western blot assay and luciferase reporter assay. Reexpression of miR-214 in HCC cell lines (HepG2 and SMMC-7721) inhibited proliferation and induced apoptosis. Furthermore, ectopic expression of miR-214 dramatically suppressed the ability of HCC cells to form colonies in vitro and to develop tumors in a subcutaneous xenotransplantation model of the BALB/c athymic nude mice. Moreover, reintroduction of XBP-1s attenuated miR-214-mediated suppression of HCC cells proliferation, colony and tumor formation. To further understand the mechanism of the miR-199a/214 cluster down-expression in HCC, we found that thapsigargin (TG) and tunicamycin (TM) or hypoxia-induced unfolded protein response (UPR) suppresses the expression of the miR-199a/21